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REMARKS 

In the specification, the paragraphs beginning on page 5, line 18; page 5, 
line 24; page 19, line 6; and page 27, line 1 have been amended to correct minor 
editorial problems. 

Claim 24 had been previously amended to address an Examiner* s objection, 
correcting an editorial problem by deleting a duplicate "with the". Claims 25 and 
39 are currently amended to correct editorial problems. 

Claims 2-15, 20-26, 28-37, 39-41 remain in this application. Claims 1, 16- 
19, and 27 are canceled without prejudice. Claim 42 has been added 

In the current Office Action, the Examiner has not rejected or objected 
to pending claim 38. Applicants believe claim 38 to be allowable. 

351LS,C, $103 

Claims 1-3. 11, 27-28, and 31 

Claims 1-3, 11, 27-28 and 31 are rejected under 35 U.S.C §103(a) as being 
unpatentable over U.S. Patent 5,995,578 to Cullen et al (Cullen). Applicants 
respectfully traverse the rejection. Claims 1 and 27 are canceled without 
prejudice. 

Currently amended independent claim 2 recites 
A method comprising: 

initiating a search for images based on at least one query keyword in 
a query; 

identifying, during the search, first images having associated 
keywords that match the query keyword and second images that contain 
low-level features similar to those of the first images; and 
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ranking the first and second images. 



7 



2 Cullen describes a searching system that is performed on a document 

database using text search, and a grouping of documents (images) based on 
appearance of the documents. Documents of interest are first searched by a text 

5 string. Documents meeting the text string search are grouped in clusters based on 

6 similarity of features. The documents are not ranked in any particular order. 
Cullen fails to teach or suggest "ranking the first and second images" as 

8 recited by claim 2. 

9 The Examiner argues that Cullen teaches this element. Specifically, the 

10 Examiner states that "Cullen teaches ranking the first and second images ([Cullen], 
coL 6, lines 30-35)". Applicants disagree. 

Cullen describes "[d]ocuments meeting the text based search are grouped 
together in grouping step 108 to form clusters based on similarity of the features 
extracted from each image." Cullen at col. 4, lines 22-25. Cullen then describes 
that the "search continues by applying the grouping step [108] to the selected 
cluster of documents subdividing this cluster into a new set of clusters 1008, each 
having a new representative document image." Cullen at col. 6, lines 30-34, What 
the Examiner refers to in Cullen is a clustering or grouping of images based on a 
similarity of features. However, Cullen does not teach or suggest inking" of 
images or documents. 

In contrast to grouping of images as disclosed in Cullen, ranking of images 
is based on features that are more similar to a target or reference image as 
described on page 13 of the current application. A first image may be ranked 
higher than a second if it more closely matches the features of the target or 
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reference image. In grouping, the first and second images may be grouped 

2 together if they have similarities that match a condition; however, since they are 

3 not "ranked" it is undetermined as to how they compare to a target image, since 

4 they are never compared or ranked against the target image. 

s Applicants respectfully request that the §103 rejection of claim 2 be 

6 withdrawn. 

7 Dependent claims 3 and 11 depend from and comprise all the elements of 
* currently amended claim 2. As such, dependents claims 3 and 1 1 are allowable by 

9 virtue of their dependency on base claim 2. Applicants respectfully request that 

10 the § 103 rejection of claims 3 and 1 1 be withdrawn, 

\ i Currently amended independent Claim 28 recites 



12 
13 



An image retrieval system comprising: 



a query handler to handle both keyword-based queries having one or 
H more search keywords and content-based queries having one or more low- 

level features of an image; and 



a feature and semantic matcher to identify at least one of (1) first 
images having keywords that match the search keywords from a keyword- 
n based query, and (2) second images having low-level features similar to the 

low-level features of a content-based query, wherein the feature and 
18 semantic matcher ranks the images. 

J 9 Cullen does not teach or suggest the image retrieval system of claim 28. In 

20 particular, Cullen fails to teach or suggest 'the feature and semantic matcher ranks 

21 the images". 

22 The Examiner presents the same arguments in rejecting claim 28 as 

23 presented in rejecting claim 2. Applicants reassert the arguments presented above 

24 supporting claims 2, in support of claim 28. Applicants respectfully request that 

25 the § 103 rejection of claim 28 be withdrawn. 
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Dependent claim 31 depends from and comprises all the elements of claim 
28. As such, dependent claim 31 is allowable by virtue of its dependency on base 
claim 28. Applicants respectfully request that the §103 rejection of claim 31 be 
withdrawn. 

Claims 4-10, and 32-37 

Claims 4-10, and 32-37 are rejected under 35 U.S.C. §103(a) as being 
unpatentable over Cullen in view of U.S. Patent 6,369,811 to Graham et al 
(Graham). Applicants respectfully traverse the rejection. 

Currently amended independent claim 4 recites 

A method comprising: 

initialing a search for images based on at least one query keyword in 
a query; 

identifying, during the search, first images having associated 
keywords that match the query keyword and second images that contain 
low-level features similar to those of the first images; 

presenting the first and second images to a user, and 

monitoring feedback from the user as to which of the first and 
second images are relevant to the query. 

Cullen describes an initial search of documents (images) based on a text 
string. If the text based search yields the document of interest, then the user 
discontinues the search. A second search is based on a representative document 
chosen from the group of documents from the initial search, Cullen describes that 
in a preferred embodiment, features from the set of documents found during the 
text search are stored in a database. Cullen does not disclose that the set of 
documents (images) from the first search are ever presented to the user. Since the 
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first documents are not presented to the user, no monitoring of feedback as to the 
first images may be performed along with second images from the second search. 

Cullen fails to teach or suggest "presenting the first and second images to a 
user and monitoring feedback from the user as to which of the first and second 
images are relevant to the query." as recited by claim 4. 

The Examiner argues that Cullen teaches this element Specifically, the 
Examiner states Cullen as teaching '^presenting the first and second images to a 
user** (col. 4, lines 14-27). However, Cullen actually teaches that a "representative 
document image is selected in step 110 for each cluster of document images 
formed in the grouping step 108". Cullen col. 4, lines 25-27, A single document 
is chosen from the first search to be the basis of the second feature search; 
however, the other images from the first search are not used for the second feature 
search. A single document is used for second feature search. Furthermore, the 
documents from the first search are not disclosed as presented to the user, along 
with images from the second feature based search. 

In addition, Cullen discloses that only features of documents, and not the 
documents from the first text based search, are kept in a database. Cullen col. 4, 
lines 20-22. Features of documents are not the same as the documents images 
from which they are extracted from. The features are a limited subset of the 
documents. Cullen further fails to teach or suggest that such features are ever 
presented along with images from the feature based search. 

Graham is cited for teaching 1 "monitoring the feedback from the user or the 
system". Graham, however; provides no assistance as to the recited methodology 
of claim 4, in light of the Cullen's lack of teaching or suggestion that the first 
images are presented along with the second images. If the first images are never 
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presented, a user's feedback is never monitored as to such first images. Applicants 
respectfully request that the §103 rejection of claim 4 be withdrawn. 

Currently amended independent claim 5 recites similar language as to 
claim 4 and benefits from the same arguments as claim 4. Applicants respectfully 
request that the §103 rejection of claim 4 be withdrawn. 

Dependent claims 6-10 depend from and comprise all the elements of 
7 currently amended claim 2. 

s Applicants reassert the arguments presented above supporting claim 2, in 

9 support of claims 6- 1 0* 

10 Claims 6-10 fUrther include elements of 4 'presenting the first and second 
i i and images to a user" and "receiving feedback from the user". The combination of 

12 Cullen and Graham fails to teach or suggest these elements. Applicants reassert 

13 the arguments above supporting claim 4, in support of claims 6-10, Applicants 

14 respectfully request that the § 1 03 rejection of claims 6-10 be withdrawn. 

15 Currently amended independent claim 32 recites 
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An image retrieval system comprising: 



a query handler to handle both keyword-based queries having one or 
is more search keywords and content-based queries having one or more low- 

level features of an image; 



a feature and semantic matcher to identify at least one of (1) first 
images having keywords that match the search keywords from a keyword- 

21 based query, and (2) second images having low-level features similar to the 
low-level features of a content-based query, wherein the feature and 

22 semantic matcher ranks the images; 



a user interface to present the images identified by the feature and 
24 semantic matcher to a user, the user interface allowing the user to indicate 

whether the images are relevant to the queiy; and 
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a feedback analyzer to train the image retrieval system based on user 
feedback as to relevancy. 

The combination of Cullen and Graham fails to teach or suggest the method 
of claim 32. Claim 32 includes all the elements of claim 28> and benefits from the 
arguments presented in support of claim 28; in particular the arguments presented 
as the recited element of a "feature and semantic matcher that ranks the images". 

Claim 32 further includes the elements of "a user interface to present the 
images identified by the feature and semantic matcher to a user, the user interface 
allowing the user to indicate whether the images are relevant to the query; and a 
feedback analyzer to train the image retrieval system based on user feedback as to 
relevancy". 

As discussed above in support of claim 4, the combination of Cullen and 
Graham fails to teach or suggest presenting 'images identified by the feature and 
semantic matcher", "allowing the user to indicate whether images are relevant", 
and "feedback" from the user. Applicants reassert the arguments above supporting 
claim 4, in support of claim 32. Applicants respectfully request that the §103 
rejection of claims 32. 

Dependent claims 33-37 depend from and comprise all the elements of 
currently amended claim 28. 

Applicants reassert the arguments presented above supporting claim 28, in 
support of claims 33-37, 

Claims 33-37 further include elements of presenting 'images identified by 
the feature and semantic matcher", "allowing the user to indicate whether images 
are relevant", and "feedback" from the user. Applicants reassert the arguments 
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above supporting claim 4, in support of claim 33-37. Applicants respectfully 
request that the § 103 rejection of claims 33-37 be withdrawn. 
Claims 12-15, and 39-41 

Claims 12-15, and 39-41 are rejected under 35 U.S.C. § 103(a) as being 
unpatentable over U.S. Patent 5,579,471 to Barber et al (Barber). The Examiner 
makes mention of a Lang reference; however, does not particular mention it as a 
patent or as part of a §103 rejection with Barber. Emmanuel Rivera contacted the 
Examiner and it was clarified that the Examiner refers to U.S. Patent 6,314,420 to 
Lang et al (Lang). Applicants respectfully traverse the rejection. 

Independent claim 12 recites 

A method comprising; 

permitting entry of both keyword-based queries and content-based 
queries; 

finding images using both semantic-based image retrieval and low- 
level feature-based image retrieval; 

presenting the images to a user so that the user can indicate whether 
the images are relevant; and 

conducting semantic-based relevance feedback and low-level 
feature-based relevance feedback in an integrated fashion. 

Barber describes a visual characteristic search of images in a database. 

Images may be selected based on several characteristics including color, pixel 

information, and text annotation. The images from the search are simply presented 

to a user without any feedback from the user as to relevance. This precludes the 

ability to perform semantic-based and feature-based feedback. 
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The combination of Barber and Lang fails to teach or suggest the method of 
claim 12. Barber does not suggest or teach '^presenting the images to a user so that 
the user can indicate whether the images axe relevant." 

The Examiner specifically states that "Barber teaches .presenting the 
images to a user so that the user can indicate whether the images are relevant (col. 
5, lines 30-42). 

Barber shows an image query performed by a "query-by-image-content 
(QBIC) engine 32 which receives the RUN QUERY". Barber at col. 5, lines 31- 
32. 'The results of the search are used by the QBIC engine 32 to construct a result 
list of images satisfying the query parameters ... and the images in the result list 
are provided to the window control 22 for display in a results window, which is not 
shown." Barber at coL 5, lines 34-42. Although Barber describes how the results 
are presented to the user, Barber does not disclose that the user may indicate 
whether the results are relevant. The method described in Barber would not have a 
need to allow the user to indicate which results are relevant, since as the Examiner 
admits "Barber does not explicitly teach conducting semantic-based relevance 
feedback and low-level feature-based feedback". Without a need to provide 
feedback, there is no need to provide a user the ability to indicate if the results are 
relevant. 

Lang is cited for teaching "the user give the feedback on the search results". 
Lang, however, provides no assistance in light of Barber as to the recited methods 
of claim 12. Since Barber does not provide the ability to indicate if the results are 
relevant, it would not have been obvious to combine Barber with the "user 
feedback" of Lang to provide either semantic-based relevance or feature-based 
feedback. 
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Accordingly, a combination of Barber and Lang fails to teach or suggest the 
claimed methods. Applicants respectfully request that the § 103 rejection of claim 
12 be withdrawn. 

Dependent claims 13-15 depend from and comprise all the elements of 
currently amended claim 12. As such, dependent claims 13-15 are allowable by 
virtue of their dependency on base claim 12. Applicants respectfully request that 
the § 1 03 rejection of claims 1 3- 1 5 be withdrawn. 

Independent claim 39 recites 

A computer-readable medium having computer-executable 
instructions that, when executed, direct a computer to: 

find images using both semantic-based image retrieval and low-level 
feature-based image retrieval; 

present the images to a user so that the user can indicate whether the 
images are relevant; and 

concurrently conduct semantic-based relevance feedback and low- 
level feature-based relevance feedback. 

The combination of Barber and Lang fails to teach or suggest the computer- 
readable medium of claim 39. Barber does not suggest or teach "presenting] the 
images to a user so that the user can indicate whether the images are relevant/* 
Lang does not suggest or teach "concurrently conducting] semantic-based 
relevance feedback and low-level feature-based relevance feedback". 

The Examiner presents the same arguments in rejecting claim 39 as 
presented in rejecting claim 12. Applicants reassert the arguments presented above 
supporting claims 12, in support of claim 39. Applicants respectfully request that 
the §103 rejection of claim 39 be withdrawn. 
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Dependent claims 40 and 41 depend from and comprise all the elements of 
base claim 39. As such, dependent claims 40 and 41 are allowable by virtue of 
their dependency on base claim 39. Applicants respectfully request that the §103 
rejection of claims 40 and 41 be withdrawn. 

Claims 16-19, and 38 

Claims 16-19, and 38 are rejected under 35 U.S.C. §103(a) as being 
unpatentable over U.S. Patent 5,899,999 to De Bonet (De Bonet). Claims 16-19, 
and 38 are canceled. 

Claims 20, and 22-23 

Claims 20, and 22-23 arc rejected under 35 U.S.C. § 103(a) as being 
unpatentable over U.S. Patent 6,092,080 to Gustman (Gustman). Applicants 
respectfully traverse the rejection. 

Independent claim 20 recites: 

A method comprising: 

presenting a result set of images that are returned from an image 
retrieval search of a query having at least one keyword; 

monitoring feedback from a user as to whether the images in the 
result set are relevant to the query; 

in an event that the user selects at least one image as being relevant 
to the query, associating the keyword in the query with the selected image 
to form a first keyword-image association and assigning a comparatively 
large weight to the first keyword-image association; and 

in an event that the user identifies an example image for refinement 
of the search, associating the keyword in die query with the example image 
to form a second keyword-image association and assigning a comparatively 
small weight to the second keyword-image association. 
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Gustman describes a system for cataloguing, storing, retrieving, and 
distributing multimedia data* Gustman describes use of a quality assurance 
mechanism that marks input data, not output data, with quality assurance events. 
Specifically, a user marks the input data with the quality assurance events. 
Therefore, prior to any searching, the user has marked the input data with quality 
assurance events. The quality assurance events distinguish input data from one 
another, and searching may be performed on particular quality assurance events, 
g Gustman fails to teach or suggest the method of claim 20. The Examiner 

9 admits that 



Gustman does not explicitly teach monitoring feedback from a user 
as to whether the images in the result set are relevant to the query, 
in an event that the user selects at least one image as being relevant 
12 to the query, associating the keyword associating the keyword in 

the query with the selected image to form a first keyword-image 
association and assigning a comparatively large weight to the first 
keyword-image association; and in an event that the user identifies 
an example image for refinement of the search associating the 

15 keyword in the query with the example image to form a sefcond 
keyword-image association and assigning a comparatively small 

16 weight to the second keyword-image association. 

17 The Examiner cites Gustman as teaching "a quality assurance mechanism 
* 8 can be used to monitor the quality of fee input data and provide feedback", 
w Gustman, however, fails to teach or suggest presenting a result set of images that 
20 are returned from an image retrieval search"; then "monitoring feedback from a 
*i user as to whether the images in the result set are relevant to the query'; and then 
22 "a first keyword-image association** and "a second keyword-image association" as 
2 * recited by elements of claim 20. 

24 Gustman discloses a quality assurance mechanism that marks input data, but 

25 fails to teach or suggest that the quality assurance mechanism may be used to mark 
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or associate the results from the query, Gustman particularly describes quality 
assurance events that are added to input data by a user, where the input data is to 
be searched later. Gustman at col. 17, lines 21-25. There is no teaching or 
suggestion in Gustman that output results (searched images) may be marked as 
feedback from the user as relevant to the query. Accordingly, Gustman fails to 
teach or suggest the claimed methods. Applicants respectfully request that the 
§ 103 rejection of claim 20 be withdrawn. 

Dependent claims 22-23 are allowable by virtue of their dependency on 
base claim 20. Applicants respectfully request that the §103 rejection of claims 
2 1 -23 be withdrawn. 

Claim 21 

Claim 21 is rejected under 35 U.S.C. §103(a) as being unpatentable over 
Gustman in view of Barber. Applicants respectfully traverse the rejection. 

Dependent claim 21 depends from and comprises all the elements of claim 
20, Applicants reassert the arguments presented above supporting claim 20, in 
support of claim 2L 

As such, dependent claim 21 is allowable by virtue of its dependency on 
base claim 20. Applicants respectfully request that the § 103 rejection of claims 21 
be withdrawn. 

Claims 24-25 

Claims 24-25 are rejected under 35 U.S.C. § 103(a) as being unpatentable 
over U.S. Patent 6,442,438 to Nojima (Nojima) in view of U.S. Patent 6,504,571 
to Narayanaswami (Narayanaswami). Applicants respectfully traverse the 
rejection. 
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Independent claim 24 recites 
A method comprising: 

computing, for each category, a representative feature vectors of a 
set of existing images within the category; 

determining a set of representative keywords that are associated with 
the existing images in each category, 

comparing, for each new image> the low-level feature vectors of the 
new image to the representative feature vectors of the existing images in 
each category to identify a closest matching category; and 

labeling the new image with the set of representative keywords 
associated with the closest matching category. 

Nojima describes a video information retrieval system that allows video 
data to be inputted and retrieved. For each video data that is stored (and may be 
retrieved), annotation information such as image feature vectors are used to 
identify the particular video data. The categories of the video data are not 
provided a representative feature vector. 

The combination of Nojima and Narayanaswami fails to teach or suggest 
the method of claim 24. 

The Examiner specifically cites Nojima as teaching "computing, for each 
category, a representative feature vectors of a set of existing images within the 
category." Nojima, however; fails to teach or suggest that feature vectors may be 
computed for each category. Although Nojima describes the use of feature vector, 
such feature vectors are related to individual and particular video data. Nojima 
describes the use of a feature vector calculation processing module 72 that 
calculates an image feature vector for each extracted still image. Nojima at col. 5, 
lines 10-12. Nojima does not teach or suggest that feature calculation processing 
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1 module 72 may be used for a category of images. Although Nojima is cited for 

2 teaching "comparing, for each new image, the low-level feature vectors of the new 

3 image to the representative feature vectors of the existing images in each category 

4 to identify a closest matching category", the Examiner has not shown where in 
$ Nojima it is disclosed that representative feature vectors are used to identify a 

6 closest matching category, since Nojima is directed to applying feature vectors to 

7 individual objects and not categories. 

8 Narayanaswami is cited for teaching 'the annotation module annotates the 

9 keywords to the images" Narayanaswami, however, provides no assistance as to 

10 the recited methodology of claim 24. Applicants respectfully request that the §103 

1 1 rejection of claim 24 be withdrawn. 

12 Dependent claim 25 depends from and comprise all the elements of claim 

13 24. As such, dependent claim 25 is allowable by virtue of its dependency on base 

14 claim 24. Applicants respectfully request that the §103 rejection of claim 25 be 
is withdrawn. 

16 Claim 26 

n Claim 26 is rejected under 35 U.S.C. § 103(a) as being unpatentable over 

is Nojima m view of Narayanaswami and in further view of U.S. Patent 5,594,809 to 

io Kopec etal (Kopec). Applicants respectfully traverse the rejection. 

20 Dependent claim 26 depends from and comprises all the elements of claim 

21 24. Applicants reassert the arguments presented above supporting claim 24, in 

22 support of claim 26. 

23 As such, dependent claim 26 is allowable by virtue of its dependency on 

24 base claim 24. Applicants respectfully request that the §103 rejection of claim 26 

25 be withdrawn. 
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Claims 29-30 

Claims 29-30 are rejected under 35 U.S.C. § 103(a) as being unpatentable 
over Nojima in view of Narayanaswami and in further view of U.S. Patent 
6,504,571 to Stuckey et al (Stuckey). Applicants respectfully traverse the 
rejection. 

Dependent claims 29-30 depend from and comprise all the elements of 
amended claim 28. Applicants reassert the arguments presented above supporting 
claim 28, in support of claims 29-30. 

The Examiner has not shown where in Nojima or Narayanaswami it is 
disclosed "a feature and semantic matcher to identify at least one of (1) first 
images having keywords that match the search keywords from a keyword-based 
query, and (2) second images having low-level features similar to the low-level 
features of a content-based query, wherein the feature and semantic matcher ranks 
the images" as recited in base claim 28. Stuckey provides no assistance as to the 
image retrieval system of claims 29-30, 

According a combination of Nojima, Narayanaswami, and Stuckey fails to 
teach or suggest the image retrieval system of claims 29-30. Applicants 
respectfully request that the §103 rejection of claims 29-30 be withdrawn. 
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CONCLUSION 

All pending claims 2-15, 20-26, 28-37, 39-41 are in condition for 
allowance. Applicant respectfully requests reconsideration and prompt issuance of 
the subject application. If any issues remain that prevent issuance of this 
application, the Examiner is urged to contact the undersigned attorney before 
issuing a subsequent Action. 



Dated: tt>f*fc3 By: 



Respectfully Submitted, 

<4 ~ 



Emmanuel A. Rivera 
Reg. No. 45,760 
(509)324-9256 ext. 245 
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